Comparison between multistage filters and sketches for finding heavy hitters

The purpose of this write-up is to compare multistage filters [3] and sketches with respect to their ability to identify heavy hitters. In a nutshell, the conclusion is that multistage filters as I use them identify heavy hitters with less memory than sketches, but some sketches support important other operations, more specifically they can be added and subtracted without any need to re-read the data stream(s). Both multistage filters and sketches work in the streaming model: data items with various identifiers make up a stream of traffic or updates that the data structures operate on. Both hash these updates to counters based on the identifiers. The main difference between sketches and multistage filters is that sketches see these counters as a summary of the traffic that can be used for many operations while filters use these counters only as a means for identifying the heavy hitters. After discussing the differences in more detail, I will compare the memory usage of these solutions for the heavy hitter problem. Since there are many types of sketches and there are many things they are used for, I am going to focus on three papers that use sketches for detecting heavy hitters. [1] is a widely quoted paper that addresses exactly the problem of finding heavy hitters. [2] is a more recent paper that proposes improved sketches for finding heavy hitters. [4] applies sketches to detect big changes in network traffic, which is related to but different from finding heavy hitters. One thing worth pointing out is that many of the sketch papers papers (and all three discussed here) are concurrent with or published after my paper on multistage filters [3].